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Expert  System  technology  has  in  recent  years  advanced  to  the 
point  where  it  is  being  increasingly  viewed  as  a  practical  tool  for 
supporting  decision  makers  addressing  significant  real  world 
decision  problems.  Associated  with  this  trend,  however,  has  'been 
some  concern  about  the  nature  of  user  interaction  with  expert 
systems  and  understanding  the  conditions  under  which  cooperative  and 
effective  user/expert  system  problem  solving  behavior  can  occur. 

This  paper  describes  empirical  research  investigating  the 
cognitive  psychology  of  user  interactions  with  expert  systems.  This 
research  was  performed  in  the  context  of  a  more  general  research 
program  that  focuses  on  achieving  two  interrelated  goals:  (1)  to 
advance  a  general  understanding  of  the  psychology  of  user 
interactions  with  intelligent  machines;  and  (2)  to  generate  design 
principles  that  lead  to  the  optimal  user  engineering  of  future 
expert  systems. 

The  central  theory  discussed  below  is  that  the  nature  of  the 
cognitive  interaction  between  a  user  and  a  rule-based  expert  system 
is  driven  to  a  significant  extent  by  two  mediating  variables: 

1.  The  degree  of  consistency  between  the  rule-based 
system  and  the  user's  problem  solving  processes; 

2.  The  user's  mental  model  of  the  system's  operating 
processes. 


The  remainder  of  this  paper  (1)  provides  a  brief  overview  of 
expert  system  technology,  (2)  discusses  some  research  relating  human 
jjv  and  expert  system  problem  solving,  (3)  presents  a  basic  theory  of 

user  interaction  with  an  expert  system,  and  (4)  provides 
;v  experimental  results  testing  this  theory. 
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Expert  Systems  in  Artificial  Intelligence 


In  the  past  decade,  a  decision  aiding  technology  has  emerged 


from  the  discipline  of  artificial  intelligence  that  has  the 


potential  for  greatly  improving  the  decision  speed  and  quality  of 


problem  solving.  Expert  systems  are  computer-based  systems  that 


provide  expert  advice  (e.g.,  medical  diagnoses)  to  users  in 


real-world  complex  problem  domains.  Expert  system  design  usually 


consists  of  two  components:  a  "knowledge  base"  and  an  "inference 


engine".  The  knowledge  base  contains  all  the  domain- relevant 


expertise.  This  expertise  is  usually  encoded  in  the  form  of 


condition-action  pairs,  referred  to  as  production  rules  that  specify 


a  set  of  heuristics  about  "What  to  do  if..."  The  inference  engine 


serves  two  functions.  First,  it  implements  general  control 


procedures  for  deciding  the  sequence  of  rules  to  test.  Second,  if  a 


mathematical  model  of  "inference  strength"  is  utilized,  the 


inference  engine  will  update  the  relevant  values  (e.g.,  likelihood 


estimates) .  In  effect,  the  role  of  the  inference  engine  is  to  apply 


domain  specific  knowledge  to  problem  specific  data  to  generate 


problem  specific  conclusions. 


In  addition  to  simply  being  encoded  as  production  rules,  a 


knowledge  base  can  usually  be  conceptually  organized  into  some  type 


of  nearly  decomposable  structure.  For  example,  as  illustrated  in 


Figure  1,  an  inference  network  is  a  structure  that  contains 


top-level  hypotheses,  called  goal  hypotheses,  which  are  decomposed 


into  various  levels  of  subhypotheses  that  are  further  broken  down 


into  specific  items  of  evidence  that  can  support  those  hypotheses. 


With  each  node,  there  is  often  an  associated  prior  degree  of  belief 
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SAMPLE  FORM  OF  AN  INFERENCE  NETWORK 


and  a  rule  for  combining  subnode  degree  of  belief  values  into  an 
updated  degree  of  belief  for  the  node. 

Most  expert  systems  developed  to  date  have  functioned  in  the 
role  of  independent  consultation  systems,  where  all  problem  specific 
information  required  by  the  system  is  directly  requested  from  the 
user.  In  a  typical  session  with  an  expert  consultation  system  the 
system  attempts  to  evaluate  the  degree  of  belief  of  a  goal 
hypothesis  by  chaining  down  the  inference  net,  identifying  the 
evidence  items  that  affect  the  goal  hypothesis,  and  querying  the 
user  about  each  relevant  item  of  evidence.  Consequently,  user 
interactions  with  an  expert  system  during  a  consultation  session 
consist  primarily  of  answering  system  questions  and  occasionally 
requesting  an  explanation  of  the  inference  process  (see  Duda  et  al., 
1979;  Shortliffe,  1976). 

Some  recent  applications  of  expert  system  technology  however 
nave  focused  on  the  use  of  expert  systems  as  an  intelligent 
interface  between  a  user  and  a  larger  complex  information  processing 
system.  An  expert  interface  system  can  be  defined  as  a  software 
system  that  uses  a  rule-based  model  to  automatically  monitor  the 
contents  of  an  external  data  source,  generate  conclusions  using  the 
data  source  as  the  primary  information  source,  and  inform  the  user 
of  those  conclusions  along  with  the  data  that  lead  to  the  ultimate 
conclusions.  The  primary  function  of  the  expert  interface  system  is 
to  enhance  an  operator's  ability  to  utilize  an  independent  baseline 
system,  but  typically  does  not  provide  the  sole  interface  to  the 
baseline  system  (see  Figure  2).  (See  Lehner ,  et  al.,  1983  for 
discussion.) 

While  expert  interface  systems  implement  the  same  rule-based 


I; 


I 


program  architectures  found  in  consultation  systems,  instead  of 
querying  the  user  to  ascertain  basic  pieces  of  evidence,  the  expert 
interface  system  will  call  up  any  of  a  set  of  application  specific 
primitive  functions  to  evaluate  evidence  nodes.  These  primitive 
functions,  in  turn,  can  interrogate  (get  input  from)  external  data 
sources  (i.e.,  the  baseline  system)  for  factual  information  (see 
Figure  2) . 

From  a  human  factors  perspective,  there  are  some  key  aspects  of 
expert  interface  systems  that  make  them  very  different  from  expert 
consultation  systems.  First,  in  a  consulting  system  users  must  have 
sufficient  domain  expertise  to  answer  the  system  queries,  whereas 
expert  interface  systems  are  essentially  turn  key  systems  that 
obtain  problem  specific  data  from  other  sources.  Consequently,  the 
user  community  of  the  expert  interface  systems  will  reflect  a  very 
broad  range  of  problem  domain  expertise,  and  such  systems  must  be 
compatible  with  differing  levels  of  user  expertise.  Second,  the 
time  constraints  on  the  decision  processes  of  these  new  interface 
systems  is  typically  shorter  and  more  diverse  than  was  the  case  with 
consultation  systems.  Third,  since  the  information  inputs  used  by 
an  expert  interface  system  are  obtained  from  an  external  data  base, 
and  not  the  user,  it  cannot  be  assumed,  when  a  result  is  displayed, 
that  the  user  is  already  intimately  familiar  with  all  the  relevant 
data  that  lead  to  that  result.  A  general  theory  of  user/expert 
system  interaction  must  be  able  to  deal  with  both  types  of  expert 
systems. 
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Hunan  and  Expert  System  Problem  Solvinc 


The  processes  of  expressing  different  problem  solving  strategies 


in  rule-based  expert  systems  has  lead  to  a  consideration  of  human 


cognition  in  terms  of  similar  rule-based  models.  Rouse  (1980) 


describes  a  production  system  as  a  rank-ordered  set  of  pattern-evoked 


rules  of  action  such  that  actions  modify  the  pattern,  thereby  evoking 


other  actions.  Human  cognitive  processing  has  been  viewed  in  a 
similar  way.  In  discussing  the  application  of  production  systems  to 


the  modeling  of  human-computer  interaction,  Durrett  and  Stimmel  (1981) 
conclude  that  production  systems  may  provide  powerful  empirical  and 


theoretical  techniques  for  investigating  human  factors  issues 
associated  with  interactive  computer  systems.  Young  (1979)  considers 


the  similarity  between  production  systems  and  aspects  of  the  human 


cognitive  system  as  creating  the  potential  for  addressing  theoretical 


issues  in  cognitive  psychology. 


Given  the  above  perspective  on  human  problem  solving,  one  way  to 


view  user  interactions  with  expert  systems  is  in  terms  of  two  separate 


production  systems  working  in  tandem  to  address  a  specific  problem 


solving  domain.  Each  system,  both  human  user  and  expert  rule  base. 


bring  some  degree  of  domain  knowledge  and  problem  solving  strategies 


to  the  decision  making  environment.  However,  it  is  unlikely  that  both 


system  share  identical  data  sets,  problem  solving  heuristics  (i.e.. 


rules)  or  control  strategies.  Effective  human/machine  cooperative 


problem  solving  occurs  when  both  systems'  approaches  combine  to  form 


one  unified  strategy  that  results  in  significantly  higher  performance 


than  either  system  could  have  reached  independently. 


Basic  Theory 

A  basic  theory  is  proposed  that  relates  the  quality  of 
cooperative  problem  solving  with  an  expert  system  to:  (1)  cognitive 
consistency,  the  degree  of  consistency  between  the  rule-based  system 
and  the  user's  problem  solving  processes;  and  (2)  mental  model,  the 
user's  conceptual  understanding  of  the  basic  principle  of  the  system's 
problem  solving  processes. 

Cognitive  consistency  is  defined  as  the  degree  to  which  the  user 
and  knowledge-based  system  utilize  similar  domain  specific  production 
systems.  That  is,  the  degree  to  which  they  share  the  same  lowest 
level  data  items,  product  rules,  and  control  procedures  (see  Figure 
3)  . 

The  quality  of  a  user's  mental  model  is  defined  in  terms  of  the 
user's  understanding  of  the  general  principles  of  the  system's 
processes,  which  may  be  formulated  by  both  exposure  to  a  conceptual 
model  of  the  system  and  direct  interaction  with  the  system.  In  the 
case  of  expert  systems,  the  user's  mental  model  concerns  the  degree  to 
which  the  user  understands  (1}  that  knowledge  has  been  encoded  as 
rules,  (2)  that  rules  are  organized  by  an  inference  network  of 
relationships,  and  (3)  that  explanatory  traces  involve  chaining 
forward  (or  backward)  along  the  inference  network.  Note  that  this 
definition  explicitly  excludes  any  reference  to  the  user's 
understanding  of  the  domain  specific  rules  in  the  expert  system,  but 
rather  focuses  on  a  user's  understanding  of  the  basic  principles  of 
the  system's  processes.  To  the  extent  that  the  user  understands  the 


expert  system's  domain  specific  knowledge,  that  knowledge  is 
incorporated  into  his  own  problem  solving  procedures,  i.e.,  becomes  a 
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component  of  cognitive  consistency. 

The  basic  theory  proposed  here  is  that  in  a  user  and  expert 
system  problem  solving  situation,  performance  is  dependent  on  both 
cognitive  consistency  and  the  user's  mental  model,  and  that  there  is  a 
strong  interaction  between  two  variables  in  their  impact  on 
performance.  Figure  4  shows  the  pattern  of  this  predicted 
interaction.  Obviously  the  strongest  prediction  is  that  a  good  mental 
model  of  expert  system  processing  will  facilitate  user  understanding 
of  system  results  and  explanations  even  if  the  system  logic  is 
substantially  different  from  the  user's.  In  particular,  when  a  user 
posesses  an  accurate  mental  model,  cognitive  inconsistency  should 
result  in  better  performance  than  problem  solving  involving  a  high 
level  of  cognitive  consistency. 

Upon  initial  examination,  this  theory  may  appear 
counter-intuitive,  as  it  predicts  cognitive  consistency  leading  to  a 
performance  decrement  for  users  possessing  accurate  mental  models. 
However,  if  viewed  from  the  perspective  of  the  user  and  machine  as  two 
interacting  production  systems  then  this  prediction  is  quite 
reasonable.  In  the  high  cognitive  consistency  condition,  the 
situation  is  such  that  the  user  and  intelligent  system  are  two  nearly 
identical  production  systems  that  are  solving  a  common  problem  by 
applying  the  same  production  rules  and  control  strategy  to  the  same 
problem  specific  data.  Given  this  high  degree  of  overlap  between  the 
two  systems  there  is  little  room  for  cooperative  problem  solving  to 
improve  upon  what  either  the  user  or  machine  could  do  in  isolation. 
As  cognitive  consistency  decreases  however,  the  potential 
improvement  of  cooperative  problem  solving  over  individual  user  or 
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in  order  to 


machine  problem  increases  proportionately.  However, 
realize  this  potential,  the  user  must  still  be  able  to  effectively 
interact  with  the  system.  It  is  proposed  here  that  an  accurate  mental 
model  will  facilitate  user/expert  system  interaction,  thereby 
permitting  effective  use  of  the  system  in  the  cognitive  inconsistent 
condition. 

Described  below  is  an  empirical  test  of  this  theory. 

EXPERIMENTAL  METHOD 


Subjects 

Thirty-two  (16  male  and  16  female)  undergraduate  students  from 
the  Catholic  University  of  America  served  as  volunteer  subjects  in 
this  study.  The  mean  age  was  19.3  years  with  a  range  of  17-22  years. 
None  of  the  subjects  had  previous  experience  with  rule-based  systems 
or  computer-aided  problem  solving  tasks. 

Materials 

The  framework  used  for  development  of  the  intelligent  interface 
was  ERS,  Embedded  Rule  Based  System  (Barth,  1982) ,  which  is  in  many 
respects  similar  to  the  well  known  PROSPECTOR  system  (Duda,  et  al., 
1979) .  The  ERS  system  consists  of  an  inference  engine,  rule  base 
parser,  and  language  for  representing  rules.  Rules  in  a  text  file  are 
parsed  and  compiled  into  internal  data  structures  during  run-time 
initialization.  The  inference  engine  then  uses  these  data  structures 
to  drive  the  systems  decision  making  process.  This  process  may 
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involve  gathering  evidence  from  the  user,  as  is  usually  done  in  expert 
consultation  systems,  or  from  a  set  of  primitives  supplied  for  a 
particular  application,  or  both.  As  sufficient  evidence  is  gathered 
conclusions  or  advice  is  reported  in  the  form  of  the  degree  of  belief 
in  the  top  level,  goal  hypotheses  that  were  defined  in  the  rule  base. 
The  system  continues  gathering  evidence  and  reporting  advice,  until  no 
more  evidence  remains  to  be  gathered,  or  the  user  issues  a  quit 
command.  Written  in  Pascal,  ERS  was  developed  on  a  VAX-11/780  under 
the  UNIX  operating  system,  version  4.1  bsd.  and  has  been  installed  on 
an  IBM  PC  with  128K  bytes  of  memory,  under  the  UCSD  P-System. 

For  this  study,  a  simplified  version  of  ERS  was  implemented  on  an 
Apple  lie  microcomputer  with  64K  bytes  of  memory.  The  rule  base 
representing  the  testbed  domain  consisted  of  an  inference  network 
containing  63  nodes,  5  goal  hypotheses,  39  rules,  and  109  links 
between  nodes. 


Experimental  Design 

A  2  x  2  factorial  design  was  used  to  create  the  experimental 
conditions.  The  two  independent  variables  were  (1)  cognitive 
consistency  and  (2)  mental  model  (see  Table  1) .  The  two  levels  of 
cognitive  consistency  were  created  by  the  nature  of  the  problem 
solving  style  used  by  the  subject.  In  each  problem,  the  expert  system 
operates  in  a  goal-driven,  backward-chaining  manner  through  the  rule 
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Group 

Number 


TABLE  1 


EXPERIMENTAL  DESIGN  MATRIX 


Independent  Variables 


I 


Mental  Model 


Accurate 


Accurate 


Cognitive  Consistency 


High 


None 


None 


High 


forward-chaining  process.  The  application  of  both  procedures  resulted 
in  identical  final  solutions  for  all  data  possiblities. 
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The  two  levels  of  the  second  independent  variable,  mental 
model,  were  (1)  accurate  mental  model  and  (2)  no  mental  model. 
Subjects  in  the  accurate  mental  model  condition  received  as  part  of 
their  instructions  a  written  description  of  an  inference  network. 
This  section  described  the  structure  of  a  general  inference  network, 
explained  how  the  expert  system  identified  goals,  intermediate 
hypothesis,  and  data  items,  and  chained  up  and  down  the  network  to 
obtain  degree  of  certainty  values  for  each  goal.  Included  in  this 
section  was  a  pictoral  display  of  an  inference  net  and  a  simple 
example  of  its  operation.  By  working  through  this  section  the  user 
developed  a  mental  model  of  how  the  expert  system  solved  problems. 

Testbed  Domain 

The  experimental  domain  was  a  simulated  stock  market  setting  in 
which  five  different  types  of  securities  fluctuated  in  value  during 
the  testing  sessions.  The  increase  or  decrease  in  security  value  was 
influenced  by  two  types  of  data:  (1)  general  market  conditions,  and 
(2)  specific  trading  activities.  Market  conditions  concerned  the 
degree  to  which  the  general  market  state  could  be  identified  as 
"bear,"  "mixed,"  or  "bull."  Trading  activities  described  the  volumes 
of  buying/selling  during  a  hypothetical  time  frame,  e.g.,  "Blue  Chip 
securities  were  sold  by  5000  shares  in  two  weeks.” 

Task  problems  were  constructed  by  creating  patterns  of  specific 
market  conditions  and  trading  activities  and  defining  the  resultant 
security  value  fluctuations.  For  each  possible  pattern  of  data 
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combinations  there  was  exactly  one  of  the  five  securities  whose 
value  increased  the  highest.  Thus,  for  each  task  problem  there  was 
one  optimum  security  that  should  be  recommended  for  purchase. 

Expert  System.  The  expert  system  included  an  inference  network  in 
which  each  of  the  five  securities  was  set  as  a  goal.  The  experimenter 
provided  the  system  with  access  to  data  concerning  previous  market 
conditions  (e.g.  "bear"),  current  market  conditions  (e.g.,  "mixed"), 
and  the  volumes  and  direction  that  each  security  is  currently  being 
traded.  The  system  utilized  backward-chaining  procedures  to  validate 
the  degree  of  belief  in  each  of  the  lowest-level  data  nodes,  assess 
value  to  the  intermediate  hypotheses,  and  in  turn  estimate  the  degree 
to  which  each  of  the  five  securities  should  be  recommended  for 
purchase.  At  this  point  the  system  displayed  the  five  securities  as 
rank-ordered  (greatest  to  poorest)  recommendations  for  purchase  (see 
Figure  5)  for  a  typical  screen  display  of  the  system's  results  for  a 
specific  problem) . 

Procedure 

After  reading  a  description  of  the  experiment  and  signing  a 
consent  form,  subjects  received  the  instruction  booklet  pertaining  to 
their  particular  mental  model/cognitive  consistency  group  condition. 
These  instructions  specified  the  objectives,  procedures,  and 
requirements  of  the  problem  solving  task.  Each  set  of  instructions 
previously  defined  the  level  of  mental  model  and  type  of  problem 
solving  procedures  to  be  implemented. 

Subjects  were  seated  at  a  large  table  directly  in  front  of  the 
expert  system  with  ample  space  to  arrange  their  individual  problem 
solving  sheets.  Upon  completion  of  the  experimental  booklet,  subjects 
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Investigated  goals  with  degree  o+  belief  >*  -100  are: 


It  is  advisable  to  buy  SPECULATIVE  (spec) 

Pr i or  degree  was  0.0.  Current  degree  is  3.7 


It  is  advisable  to  buy  BONOS  (bonds) 

Prior  degree  was  0.0.  Current  degree  is  1.6 


It  is  advisable  to  buy  BLUE  CHIP  (bluechip) 
Prior  degree  was  0.0.  Current  degree  is  -1.8 


It  is  advisable  to  buy  PREFERRED  (pref) 

Prior  degree  was  0.0.  Current  degree  is  -3.7 


It  is  advisable  to  buy  WARRANTS  (warr) 

Prior  degree  was  0.0.  Current  degree  is  -6.0 


******************************************************* 


AN  EXAMPLE  OF  THE  SCREEN  DISPLAY  OF  THE  EXPERT 
SYSTEM TS  RECOMMENDATIONS 


FIGURE  5 


received  three  test  problems  to  practice  use  of  their  own  procedures. 
The  type  of  data  presented  to  the  subject  was  of  the  same  nature, 
i.e.,  market  conditions  and  trading  activities,  that  the  expert  system 
utilized.  Competent  use  of  the  appropriate  styles  of  procedures, 
either  (1)  cognitive  consistent  or  rule-driven,  backward  chaining,  or 
(2)  cognitive  inconsistent  or  data-driven,  forward  chaining,  was 
reached  by  all  participants. 

At  this  time  subjects  received  practice  interacting  with  the 
expert  system.  After  viewing  the  system's  prioritized  list  of 
security  recommendations,  users  practiced  querying  the  system  for  its 
decision  rationale.  In  other  words,  the  user  saw  what  the  expert 
system  recommended  for  a  specific  problem  and  sought  to  determine 
"how"  and/or  "why"  this  particular  recommendation  was  reached. 
Through  the  use  of  a  node  description  command,  initiatecd  by  the 
entering  of  a  "d,"  carriage  return,  and  specification  of  the  node  name 
to  be  examined,  users  were  able  to  examine  important  components  of  the 
systems  logic  and  reasoning.  Figure  6  provides  a  typical  screen 
display  of  a  node  description. 

By  requesting  a  description  of  this  node,  named  "M3,"  the  user 

ascertains  that:  (1)  this  node  concerns  the  degree  to  which  "the 

previous  market  was  bull  and  the  current  market  is  bear";  (2)  the 
expert  systems  current  degree  of  belief  in  this  node  is  3.7;  (3)  there 
are  two  antecedent  nodes,  "cbear"  of  current  degree  of  belief  3.7,  and 
"pbull"  of  current  degree  of  belief  6.0;  and  (4)  there  exists  one 
antecedent  node,  "bear"  of  value  3.7.  While  this  specific  node 

description  may  provide  little  rationale  itself,  it  does  provide  a 

window  of  the  expert  system's  structure  and  suggests  paths  for  further 
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This  space  concerns  whether  or  not: 

the  PREVIOUS  market  was  BULL  and  the  CURRENT  market  is  BEAR  (m3) 


TYPE: 


not  asked 
Prior  Degree 
Current  Degree 


Antecedents 

cbear 

0.0 

3.7 

pbull 

0.0 

6.0 

Consequents 

bear 

0.0 

3.7 

SCREEN  DISPLAY  OF  NODE  DESCRIPTION 
FIGURE  6 


investigation,  namely  either  "cbear"  or  "pbull,"  that  may  provide 
insight  into  the  data  the  system  utilized.  Figure  7  provides  the  type 
of  structure  one  may  infer  from  the  above  example.  Note  that 
successful  interaction  with  the  system  involves  the  specification  of 
several  successive  node  description  commands  to  reach  meaningful 
information,  e.g.,  the  degree  of  certainty  of  the  systems  lowest  level 
data  items. 

The  experimental  process  tested  subjects  individually  for  six 
separate  problems.  A  time  constraint  of  150  seconds  was  imposed  upon 
each  task.  Pilot  studies  demonstrated  this  time  allowance  as  adequate 
for  proficient  interaction  with  the  system  and  use  of  individual 
problem  solving  procedures. 

Each  of  the  problem  solving  tasks  proceeded  as  follows.  Seated 
in  front  of  the  expert  interface  system  with  individual  problem 
solving  sheets  within  easy  reach,  a  subject  viewed  the  system's  rank 
ordered  list  of  security  recommendations  for  the  current  problem.  The 
experimenter  handed  the  subject  written  data  concerning  "previous 
market"  and  "current  market"  estimations.  At  this  point  the  subject 
could  allocate  time  to  either  querying  the  system  with  the  "d" 
command,  utilizing  the  individual  problem  solving  sheets,  or  a 
combination  of  both.  After  180  seconds,  the  subject  viewed  the 
system's  recommendations  a  final  time  and  terminated  the  problem 
solving  session.  The  subject  recorded  on  an  answer  sheet  by  simply 
checking  off  the  one  or  two  securities  most  recommended  and  by  writing 
a  few  sentences  describing  why. 

At  the  completion  of  six  such  problems,  subjects  completed  a 
brief  questionnaire  assessing  reaction  to  the  problem  solving 
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experience  and  were  adequately  debriefed. 

Performance  Measures 

For  each  of  the  six  problem  solving  tasks,  one  of  the  five 
securities  had  been  evaluated  as  the  optimal  recommendation  during 
task  construction.  This  benchmark  solution  was  reached  by  utilizing 
problem  solving  procedures  applied  to  the  complete  data  set.  It 
should  be  noted  that  both  types  of  procedures,  forward-chaining  and 
backward  chaining,  reach  the  same  conclusions  given  the  same  data. 
Thus,  for  each  of  the  six  problems  we  constructed  a  benchmark  solution 
as  a  comparison  for  the  subjects'  responses. 

The  major  performance  measurement  was  the  number  of  times  a 
subject's  response  matched  the  predetermined  optimal  one.  Individual 
scores  could  range  from  0,  none  of  the  problems  correct,  to  6,  all  of 
the  solutions  matched  with  the  optimal  ones. 

A  second  performance  measure  was  the  10- item  subjective 
questionnaire.  Subjects  indicated  on  a  10-point  scale  from  0  ("very 
strongly  disagree")  to  10  ("very  strongly  agree")  their  agreement  with 
statements  addressing  (1)  the  understanding  of  the  expert  interface 
system's  operating  procedures,  (2)  the  ease  of  system  use,  (3)  the 
confidence  of  final  user  decisions,  and  (4) ,  the  adequacy  of  the  time 
allotment . 

Finally,  the  number  of  user  queries  to  the  expert  interface 
system  were  recorded.  These  "d"  commands  were  noted  for  each  subject 
over  each  of  the  six  problems. 


RESULTS 


The  principle  issue  in  this  experiment  was  the  combined  effect  of 
mental  model  and  cognitive  consistency  on  subjects  problem  solving 
with  an  expert  interface  system.  Table  2  provides  the  means,  standard 
deviations,  and  range  for  each  of  the  four  experimental  groups.  The 
data  set  was  subjected  to  a  2  x  2  analysis  of  variance  procedure 
investigating  the  main  effect  of  each  of  the  independent  variables  a 
well  as  the  interaction  between  them  (Table  3) .  The  strong  main 
effect  of  mental  model,  F(l,28)  *  11.15,  p  -  .00214,  demonstrated  that 
an  effective  understanding  of  the  system's  operating  procedures 
facilitated  cooperative  problem  solving  quality.  As  predicted,  a  main 
effect  for  cognitive  consistency  did  not  occur,  F(l,28)  *  0.0.  The 
presence  of  a  significant  interaction  between  mental  model  and 
cognitive  consistency,  F(l,28)  *  8.2,  p  «  .0079,  confirmed  our  central 
hypothesis  and  theoretical  basis  for  user/system  problem  solving. 

A  graphical  presentation  of  subject  correct  responses  as  a 
percentage  of  total  problems  is  depicted  in  Figure  8.  The  mental 
model/cognitive  consistency  interaction  is  easily  observable. 
Individual  comparisons  confirmed  several  hypotheses.  For  those 
possessing  an  accurate  mental  model,  inconsistent  (forward-chaining) 
procedures  led  to  a  significantly  greater  performance,  than  consistent 
(backward-chaining)  procedures  t(14)  =  2.17,  p  <  .05.  For  users 

without  an  accurate  mental  model  performance  improved  when  consistent 
(backward-chaining)  procedures  were  followed,  although  this  difference 
failed  to  reach  significance,  t(14)  =  1.90,  p  *  .079.  When  evaluating 
the  two  groups  implementing  inconsistent  procedures,  subjects 


TABLE  2 

MEANS,  STANDARD  DEVIATIONS,  AND  RANGES  FOR  FOUR  GROUPS 


Group 

Mean 

SD 

Range 

Accurate  MM/Consistent 

3.375 

1.4079 

1-5 

Accurate  MM/lnconsistent 

4.875 

1.3562 

2-6 

No  MM/Consistent 

3.125 

1.3562 

1-5 

No  MM/lnconsistent 

1.625 

1.7678 

0-4 

possessing  an  accurate  mental  model  performed  significantly  better, 
t (14)  -  4.13,  p  <  .01. 

As  shown  in  Figure  9,  there  was  a  mental  model/cognitive 
consistency  interaction  predicted  for  each  of  the  six  test  problems. 

Further  data  analysis  was  performed  by  evaluating  subjects 
reponses  to  the  10-item  subjective  questionnaire.  Users  receiving 
accurate  mental  models  reported  greater  "understanding  of  the  system's 
operating  procedures,"  means  of  5.7  and  5.7  (cognitive  inconsistent 
and  cognitive  consistent  respectively)  than  those  without  an  accurate 
mental  model,  means  of  3.4  and  3.8  respectively.  Reports  of  "ease  of 
system  use"  followed  the  general  interaction  pattern,  the  means  being 
7.9,  6.7,  6.8,  and  5.8.  "Confidence  of  final  user  decisions"  followed 
the  same  pattern  with  means  of  7.9,  6.7,  6.8,  and  5.8.  The  "adequacy 
of  the  time  allotment"  revealed  the  lowest  performing  group  of  no 
mental  model/cognitive  inconsistent  to  be  most  time  pressured,  mean  of 
2.8,  compared  to  the  other  three  groups  with  means  of  4.8,  5.5,  and 
4.4  respectively. 

The  final  performance  measure  of  the  number  of  user  queries 
during  user/system  interaction  revealed  no  differences  across  the  four 
conditions,  the  means  being  5.4,  5.0,  5.3,  and  5.4  respectively. 
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DISCUSSION  AND  CONCLUSION 


Overall,  the  results  of  this  study  support  the  basic  theory  that 
the  quality  of  user/expert  system  cooperative  problem  solving 
performance  is  driven  to  a  significant  extent  by  both  the  degree  of 
consistency  between  the  user's  and  expert  system's  problem  solving 
procedures  and  the  user's  mental  model  of  the  expert  system's  problem 
solving  processes.  Highest  problem  solving  performance  was  reached  by 
users  possessing  a  good  mental  model  and  utilizing  forward-chaining 
(cognitive  inconsistent)  procedures.  Our  basic  explanation  for  this 
result  is  that  a  user  with  an  accurate  mental  model  is  in  a  position 
to  effectively  interact  with  the  expert  system  despite  significant 
differences  between  user's  and  expert  system's  problem  solving 
procedures.  Consequently,  such  a  user  is  in  a  position  to  exploit  the 
fact  that  the  user  and  expert  system  have  differing  capabilities  and 
areas  of  expertise.  A  user  that  both  has  inconsistent  problem  solving 
procedures  and  lacks  an  accurate  mental  model  is  not  in  a  position  to 
understand  the  expert  system's  procedures.  Poor  performance  is  due, 
in  this  case,  to  an  inability  to  successful,  interact  with  the  expert 
system.  Our  results,  therefore,  indicate  that  problem  solving 
requires  the  user  to  be  proficient  in  both  (1)  interacting  with  the 
expert  system  and  (2)  utilizing  successful  individual  problem  solving 
procedures. 

The  questionnaire  analysis  was  useful  in  several  respects. 
First,  the  groups  in  the  "accurate"  mental  model  conditions 

subjectively  reported  greater  understanding  of  the  system's  processes 
as  would  be  expected.  Second,  reported  confidence  of  final  user 


decisions  followed  the  same  interaction  pattern  indicating  that  high 
performance  and  user  confidence  went  hand-in-hand. 

There  were  no  significant  differences  in  the  actual  number  of 
user  queries  to  the  expert  system  during  the  task  sessions.  An 
explanation  for  this  may  be  that  users  possessing  an  accurate  mental 
model  did  not  increase  the  quantity  of  node  description  commands , 
rather,  they  simply  requested  information  that  would  be  of  ultimate 
use  to  them.  These  requests  may  have  focused  on  high-quality  queries, 
as  opposed  to  users  who  did  not  understand  the  expert  system's 
structure  and  obtained  often  irrelevant  information  to  the  immediate 
task  at  hand. 

From  the  perspective  of  the  potential  practical  implications  of 
this  theory,  the  most  immediate  impact  is  what  the  theory  suggests 
about  how  user  interactions  with  expert  consultation  and  expert 
interface  systems  will  differ.  Users  of  expert  interface  systems  are 
likely  to  be  significantly  inconsistent  from  the  expert  system  in  both 
the  problem  specific  data  they  are  initially  aware  of  and  the  domain 
specific  heurisitcs  utilized  in  problem  solving.  Consequently, 
user/expert  interface  system  interaction  is  a  situation  that  naturally 
relfects  a  great  deal  of  cognitive  inconsistency.  As  a  result, 
creating  an  accurate  mendal  model  may  be  an  essential  ingredient  for 
the  successful  transfer  of  interface  system  to  operational  use. 

Finally,  regarding  the  completeness  of  the  above  research  as  a 
test  of  the  theory,  it  should  be  recognized  that  this  initial  research 
has  operationalized  cognitive  consistency  as  the  match  between  the 
user's  and  the  expert  system's  procedures.  While  we  expect  the  same 
pattern  of  performance  to  result,  other  dimensions  of  cognitive 


consistency  need  to  be  examined.  Furthermore,  a  node  description 
command  was  the  only  type  of  explanantion  a  user  could  receive  in  this 
study.  This  was  chosen  primarily  because  of  the  imposed  time 
constraint  and  the  nature  of  the  task  setting.  Other  explanation 
capabilities  should  be  examined,  including  a  rule-trace  or 
presentation  of  the  system's  intermediate  hypotheses.  Finally,  user 
groups  of  diverse  expertise  levels  should  be  studied  over  several 
domains  and  under  a  varying  range  of  time  constraints. 

Ongoing  research  is  currently  addressing  further  issues  in  the 
user/system  interface  in  an  attempt  toward  developing  a  more  complete 
set  of  empirically-tested  theoretical  principles  of  user/expert  system 
interaction . 
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